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The MA Entrance Examinations (MAEE) held in Iran since 1990 are 
frequently criticized as being invalid, unstandardized exams with lots of 
problem in tenns of principles of testing in general and test construction 
in particular (for instance, Jafarpur, 1996). To make sound judgments 
about such objections, the present study dealt with a content analysis of 
the TEFL MAEE held in 2007. Actually, the purpose of this study was 
two-fold. First, it aimed at analyzing the content of the MAEE’s in order 
to see if any pattern be at work in the process of devising such exams. 
Naturally, through such an analysis, some problems of these exams were 
also determined. Thus, the second aim of the present study was to 
pinpoint and describe the problems with these exams as well as to offer 
some suggestions to remedy the problems. In so doing, a coding system 
encompassing a checklist of the possible content categories was 
developed by the researchers themselves and its validity and reliability 
were established as well. The finding of such an analysis, especially the 
unequal distribution of the content categories, supported the idea that the 
validity of the exam is not strongly established due to the exclusion of or 
de-emphasis over the content categories given significant credit in the 
B.A. Program. The problems found during the analysis showed that the 
exam is not a standard one; still some of the basic principles of language 
testing are not observed in the process of constructing the exam. 
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1 Introduction 

Entrance examinations in general and the M.A. Entrance Examinations 
(MAEE) in particular are evidently of crucial importance since based on the 
examinees’ performance on such exams, vital decisions are made; decisions 
that do have decisive impacts directly on the examinees’ future. Thus, testees 
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are undoubtfully entitled to sit for a standard, well-developed exam with as 
few problems as possible. In fact, in most cases, one may pass or fail only 
due to his/her answers just to one or two items. 

During the past twenty years or so, different studies have been 
conducted as well as reported with respect to the shortcomings of the 
University Entrance Exams in Iran. For example, investigating the exams 
administered from 1983 to 1985, Farhady (1985) found little correspondence 
between the manner materials are taught to the students and the manner the 
students are tested on them. In a similar attempt, Yarmohammadi (1986) 
came to this conclusion that in such exams, the role of communication is, to a 
great extent, ignored. Moreover, among language skills only one skill 
(reading, namely) and among language components only grammar and 
vocabulary are tested. Jafarpur (1996; see also Jarfarpu, 1986), for instance, 
in a study critically reviewed the TEFL MAEE of 1996 and concluded that 
"the validity of that exam is under question due to the problems related to the 
items in terms of testing principles" (p.20). Razmjoo (2006) stated that the 
University Entrance Examination held in Iran during the past two decades are 
frequently criticized as being invalid, and non-standardized with lots of 
problems in terms of principles of testing in general and test construction in 
particular. Yet some of the basic principles of language testing are not 
observed in the process of constructing the exam. 

Surprisingly enough, despite all these criticisms and although the 
MAEE’s in their present format are at work since 1990, still there seems to be 
lots of problems with them. 

1.1 Statement of the problem 

In order to make sound judgments about such objections, the present study 
dealt with a content analysis of the TEFL MAEE held in 2007. Actually, the 
purpose of this study was two-fold. First, it aimed at analyzing the content of 
the MAEE’s in order to see if any pattern be at work in the process of 
devising such exams. Naturally, through such an analysis, some problems of 
these exams were also determined. Thus, the second aim of the present study 
was to pinpoint and describe the problems with these exams as well as to 
offer some suggestions to remedy the problems. 

1.2 Research questions 

The present study addressed two major questions as follow: 

1. Does the MAEE measure what it is supposed to measure? In other words, 

is there any specific objective behind including the items in the exam? 

2. Is there any balance between the emphasis given to each specific course 

(i.e. the credit assigned to each course in the undergraduate program) 
and the number of items allocated to them? 
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1.3 Significance of the study 

The findings of this study can hopefully be useful for the university teachers 
teaching to English majors at the B.A. level, the TEFL MAEE testees, and 
most for the anonymous test constructors of this exam. Moreover, the 
assessment organization will benefit from the results of the study. 

1.4 Information about the exam 

The exam consists of two main parts: General English and Major English. 
The general English exam includes the following parts (Table 1). 


Table 1. The General English Exam Format 


Section 

Sub-sections 

Number of Items 

1. Structure and 

1.1 Sentence Completion 

10 items 

Written Expressions 

1.2 Error Recognition 

10 items 

2. Vocabulary 


10 items 

3. Cloze Test 


15 items 

4. Reading Comprehension 

25 items 

The major English exam consists of the following sub-majors, namely, 

Teaching Methodology, Testing and Linguistics (Table 2). 

Table 2. The Major English Exam Format 


Majors 

Sub-parts 

Number of Items 


1.1. Teaching Methodology 

40 Items 

1. TEFL 

1.2. Testing 

20 Items 


1.3. Linguistics 

20 Items 

2. English Literature 

No-subparts 

80 Items 


3.1. Theoretical principles of 



translation 



3.2. Linguistics 

15 Items 


3.3. Contrastive Analysis 

10 Items 


3.4. Morphology 

15 Items 


3.5. Translation skill 

15 Items 


2. The Study 

Three major subparts (Teaching Methodology, Testing and Linguistics) of 
the TEFL MAEE administered in 2007 were selected to obtain a 
representative sample of the test at issue. The booklet included in itself 40 
items for Methodology, 20 ones for Linguistics and 20 for Testing to keep the 
total number of items equal to 80 multiple choice items as usual. 

Next, to define content categories that measure the variables in question, a 
coding system encompassing a checklist of possible content categories was 
developed by the researchers themselves. To ensure the validity of such a system, 
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first content categories were determined based on a classification proposed by 
Mirzaeian (1997), a classification which has proved to be helpful in classifying 
test items of the MAEE’s of seven successive years. Furthermore, to have the 
checklist as comprehensive as possible, it was modified according to the 
following coursebooks which are the dominant major ones taught in the related 
courses at the B.A. Program in our country: 

Methodology 

Brown, H. D. (2000). Principles of language learning and teaching (4 ,h 
ed.). White Plains, NY: Pearson Education. 

Chastain, K. (1988). Developing second language skills: Theory’ and practice 
(3 rd ed.). New York: Harcourt Brace Jovanovich Publishers. 
Larsen-Freeman, D. (2000) Techniques and principles in language teaching. In R. 
N. Campbell & W. E. Rutherford (eds.), Teaching Techniques in English 
as a Second Language (pp. 1-189). Oxford: Oxford University Press. 
Richards, J. C., & Rodgers, T. S. (2001). Approaches and methods in 
language teaching (2 nd ed.). Cambridge: Cambridge University Press. 

Testing 

Faihady, H. et al. (1994). Testing Language skills: From theory to practice. Tehran: SAMT. 
Flarris, D. P. (1969). Testing English as a second language. New York: Me Graw-Hill. 
Fleaton, G. B. (1990). Writing English language tests. London: Longman. 
Flughes, A. (2003). Testing for language teachers. Cambridge : Cambridge 
University Press. 

Linguistics 

Falk, J. (1978). Linguistics and language. New York: John Wiley. 

Fromkin, V., Rodman, R., & Hymes, N. (2003). An Introduction to language 
(7 th ed.). Fort Worth: Holt, Rinehart and Winston, Inc. 

Richards, J. C„ Platt, J., & Platt, H. (2002). Longman dictionary’ of language 
teaching & applied linguistics (3 rd ed.). Essex: Longman. 

Yule, G. (2006). The study of language (3 rd ed.). Cambridge: Cambridge 
University Press. 

As a pilot study, the modified version, then, was applied by the 
researchers themselves to the TEFL MAEE of 2006 to improve it further. Thus, 
the researchers came up with a checklist consisting of 31 content categories for 
Methodology, 17 ones for Linguistics and 18 for Testing. Still, to reserve the 
flexibility of the checklist, for each part a choice titled “others (name, if any)” 
was added to the list. Moreover, enough room was given for possible 
comments and notes for each item. 

After establishing the validity of the content classification system, it was 
investigated to determine the reliability of the coding scheme in the sense of 
intra-coder and inter-coder reliability. For intra-coder reliability, the researchers 
codified the test items with a two-week time interval and the correlation between 
the findings was computed. To gain inter-coder reliability, two independent 
coders who were sufficiently trained, skilled and familiar with the task codified 
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the test items. The codification was done by the raters independently at the same 
time and the correlation of codifying the test items was calculated. The intra-and 
the inter-coder reliability were shown to be .97.5 and .96 respectively. 

3 Results and Discussion 

Based on the instruction proposed by Ary et al. (1996, p. 486), the materials were 
analyzed according to the content classification system to determine the content 
categories of the exam. 

To summarize the data and to aid in its interpretation, frequencies as well 
as percentages are used. Moreover, to present a clearer picture of the data, three 
bar graphs are presented. 


Table 3. Frequency and Percentage of Methodology Items 


Categories 

Frequency 

Percentage 

1. The history of language Teaching 

0 

0% 

2. The Brain and Learning 

2 

5% 

3. Grammar Translation Method 

0 

0% 

4. Direct Method 

0 

0% 

5. Situational Language Teaching 

0 

0% 

6. Audiolingual Method 

0 

0% 

7. Total Physical Response 

0 

0% 

8. Community Language Learning 

0 

0% 

9. Silent Way 

0 

0% 

10. Suggestopedia 

3 

7.5% 

11. Natural Approach 

1 

5% 

12. Functional-Notional Approach 

0 

0% 

13. Communicative Language Teaching 

3 

7.5% 

14. Cognitive Code Method 

1 

2.5% 

15. Task-Based Language Teaching 

1 

2.5% 

16. Content-based teaching 

2 

5% 

17. Cooperative learning 

1 

2.5% 

18. Multiple Intelligences 

1 

2.5% 

19. Post-method Era 

0 

0% 

20. Listening 

0 

0% 

21. Speaking 

0 

0% 

22. Reading 

2 

5% 

23. Writing 

1 

2.5% 

24. Syllabus Design 

0 

0% 

25. Drills and Class Activities 

3 

7.5% 

26. First Language Acquisition & Language Teaching 

0 

0% 

27. Theories of Second Language Learning & Teaching 

10 

25% 

28. Learning Strategies and Styles 

3 

7.5% 

29. Personality and socio-cultural Factors 

0 

0% 

30. Errors 

3 

7.5% 

31. Contrastive Analysis 

2 

5% 

32. Others (name, if any) 

1 

2.5% 

31. Total 

40 

100% 
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Table 3 indicates that equal weight is not given to the thirty-one 
parts of methodology section. Modern methods and approaches such as 
communicative language teaching and suggestopedia and theories of 
second language learning and teaching have the highest frequencies and 
percentages. No question has been raised with respect to categories such as 
the Grammar-Translation and the Direct, the audio-lingual, situational 
language teaching Methods which are the dominant traditional methods and 
the instructors spend lots of time on teaching them. Figure 1 displays the 
data. 

Figure 1. Display of methodology items 



Topmethod 
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Table 4. Frequency and Percentage of Linguistics Items 


Categories 

Frequency 

Percentage 

1. History/Origin of Language 

0 

0% 

2. Writing Systems 

0 

0% 

3. Language Properties 

2 

10% 

4. Phonetics 

2 

10% 

5. Phonology 

2 

10% 

6. Morphology 

0 

0% 

7. Syntax 

5 

25% 

8. Semantics 

3 

15% 

9. Pragmatics 

1 

5% 

10. Discourse Analysis 

0 

0% 

11. Language & the Brain 

2 

10% 

12. Language & Machines 

0 

0% 

13. Language, Society & Culture 

1 

5% 

14. Animals & Human Language 

0 

0% 

15. Language, History & Change 

1 

5% 

16. First Language Acquisition 

1 

5% 

17. Second Language Acquisition/ Learning 

0 

0% 

18. Others (name, if any) 

0 

0% 

19. Total 

20 

100% 


The same as Methodology section, equal value has not been given to 
different parts of the Linguistics section. The frequency and percentage of 7 
parts of this section is zero. Parts such syntax, semantics, phonetics and 
phonology have the highest frequencies. Figure 2 displays the linguistics items. 

Figure 2. Display of the linguistics items 
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Table 5. Frequency and Percentage of Testing Items 


Categories 

Frequency 

Percentage 

1. Test, Measurement and Evaluation 

0 

0% 

2. Test Functions 

1 

5% 

3. Test Forms 

0 

0% 

4. Test Construction and Administration 

5 

25% 

5. Validity 

1 

5% 

6. Reliability 

5 

25% 

7. Practicality 

0 

0% 

8. Fistening 

0 

0% 

9. Speaking 

0 

0% 

10. Reading 

1 

5% 

11. Writing 

0 

0% 

12. Vocabulary 

1 

5% 

13. Structure 

0 

0% 

14. Pronunciation 

0 

0% 

15. Descriptive Statistics 

0 

0% 

16. Inferential Statistics 

0 

0% 

17. Theories of Fanguage Testing 

4 

20% 

18.Cloze & Dictation 

1 

5% 

19. Others (name, if any) 

1 

5 

20. Total 

20 

100% 


The items in this section are not equally distributed among the 
content categories. Test construction, reliability and theories of language 
testing have received the highest frequency and as a result 70% of the items 
are allocated to these topics. Figure 3 displays the testing items. 


Figure 3. Display of the testing items 
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As for the choice distribution, the following tables present the choice 
distribution of each course in particular and the overall major in general (this 
is done based on the assessment organization key to the items). 


Table 6. Teaching Methodology Items Choice Distribution 


Choices 1(A) 

2(B) 

3(0 

D(4) 

Times as correct answers 13 

10 

9 

8 

It can be claimed the correct responses are distributed well among the four options. 

Table 7. Language Testing Items Choice Distribution 


Choices 1(A) 

2(B) 

3(C) 

D(4) 

Times as correct answers 0 

8 

4 

8 


The Table presents the fact that the correct responses are not distributed 
normally among the four alternatives. 


Table 8. Linguistics Items Choice Distribution 


Choices 

KA) 

2(B) 

3(C) 

D(4) 

Times as correct answers 

5 

5 

6 

4 


Based on the results of table 6, it can be claimed that there is a normal 
distribution of the correct responses of the items in linguistics course. Table 9, 
shows the choice distribution of the correct response of the whole major. 


Table 9. Choice Distribution of All Items 


Choices 

KA) 

2(B) 

3(C) 

D(4) 

Times as correct answers 

18 

23 

19 

20 


Despite the fact that the distribution of the correct responses of each 
individual course is not too much equal, the overall choice distribution is close to 
normal. 

According to Farhady (1994), in the case of multiple choice items, 
“correct responses should be distributed approximately equally but randomly 
among the alternatives” (p. 97). In this exams, the correct answers (N=80) are 
approximately distributed equally and of course randomly. However, a close look 
over the way the correct responses are distributed in each subpart of the exam 
reveals that the distribution is not equal. Making such distributions equal even for 
the subparts of the exam is a step towards the improvement of the exam. 

As for the second aim of the present study, the researchers pinpointed, 
through the thorough analysis of the exam, the following problems: 

1. It is usually suggested by the experts in the field of language 
testing that “the stem should be quite clear and state the point 
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to be tested unambiguously.” (Farhady et al., 1994, p. 93). 
However, in 2007 TEFL MAEE, there are lots of items where 
this criterion is not observed. For instance, in the 
Methodology Section, the stems of items 82, 84, 86, 90, 93, 

96, 105 and 117 are not clear. The same is true about items 
121. 122, 127, 129, 133, 135 and 136 of the Testing Section 
as well as items 143 and 149 of the Linguistics Section. 

2. Since the number of items is very limited, especially in the 
case of the Linguistics and Testing Sections, and since the 
number of the content categories, which are supposed to be 
included in the exam, is relatively high, it would be 
reasonable to test the same theme just once within a single 
MAEE. In other words, it is not justifiable that in this very 
exam, 5 items are related to syntax while no item is allocated 
to word-formation theme. 

3. Farhady et al. (1994, p. 95) state that "every item should have 
one correct or clearly best answer". Items 86 and 129 have 
more than one correct answer. 

4. The following items are not familiar to the MA candidates 
and are not included in the content MA sources: items 83, 91 
and 97 of the Methodology section, item 125 of the Testing 
section and items 156 and 160 of the Linguistics section. 

5. Due to the facts that the testees are not supposed to use any 
kind of calculators during the exam session, items 124, 131 
and 136 of the testing section require complicated 
calculations. This is not fair and justifiable. 

4 Conclusions 

As far as the first research question is concerned, it seems that the validity of 
the exam is not strongly established due to the exclusion of or de-emphasis 
over the content categories given significant credit in the B.A. Program. 
Moreover, the unequal distribution of the content categories also supported a 
negative answer to the second research question too. The problems found 
during the analysis showed that the exam is not a standard one; still some of 
the basic principles of language testing are not observed in the process of 
constructing the exam. The only solution to remove such problems and 
shortcomings is to employ experts in the field as exam developers. 

The present study is limited in its scope: only one of the MAEE’s 
was analyzed; no attempt was made to determine the exact sources from 
which the items were produced. Nothing can be said about the reliability of 
the exam due to the unavailability of the scores too. One can replicate this 
research study focusing on any kind of correspondence between the content 
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of the MAEE and that of the MA courses. The researchers hope that the 
findings of this very study and other similar ones help improve the MAEE’s 
since evaluation is not to prove but to improve. 
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